An important feature of virtualized systems is an ability to “migrate” a virtual machine (VM) running on one host computer to another host computer. The migration may be needed, for example, for load balancing. Prior to migration, the current state of the VM on a source host computer is captured and a “snapshot” of the VM is created. The snapshot includes the then-current state of the VM's CPU, RAM and peripheral devices (e.g., network adapter, storage drives, etc.). The snapshot is then migrated or transferred to a destination host computer, and is used to instantiate a new VM which is able to resume operations without loss of data or state. Capturing the current state of the VM is possible because the VM's hardware (i.e., CPU, RAM, peripheral devices, etc.) are emulated in software, and therefore, data that characterize the state of the VM can be readily extracted from the RAM of the source host computer.
As a way to improve I/O performance of emulated devices, manufacturers of I/O devices, such as storage and network devices, have begun to incorporate virtualization support directly into the device. For example, an I/O device that complies with the Single Root I/O Virtualization (SR-IOV) specification enables multiple VMs on a single host computer to share resources of the device. Specifically, virtualization software, for example, a hypervisor, of a virtualized host computer can instruct device drivers of an SR-IOV compliant peripheral device to partition and allocate the device's physical resources (e.g., registers, RAM, etc.) to support simultaneously multiple VMs running on the host computer.
When the host computer includes a Memory Management Unit (MMU) as part of its hardware architecture, the virtualization software, for example, the hypervisor, is able to provide device drivers of a VM's guest operating system (OS) mappings between the VM's virtual addresses used by the device drivers of the guest OS to transmit instructions to devices through memory mapped I/O techniques and machine addresses of the host computer that have been mapped to I/O control registers within the corresponding physical devices. These MMU mappings enable the device drivers of a guest OS to transmit device instructions directly to their corresponding physical devices by writing such instructions into addresses of the VM's own virtual address space that have been allocated to the physical device through memory mapped I/O. When the host computer additionally includes an I/O Memory Management Unit (IOMMU) as part of its hardware architecture, the virtualization software is further able to provide physical devices mappings of a VM's physical addresses, as understood by the VM's guest operating system (OS), to their corresponding machine addresses. These mappings enable, for example, an SR-IOV compliant device that has directly received an instruction from a device driver of a guest OS (e.g., via memory mapped I/O and mappings in the MMU) that includes references to the VM's physical addresses as understood by the guest OS to read and write directly from and to the VM's address space, in accordance with the received instruction, thereby obviating a need for an emulated device that assists with such physical-to-machine address translations. Physical devices that are exposed directly to the guest OS through a MMU and can directly access the VM's address space in the host computer system machine memory through an IOMMU in the manner described above are known as pass-through devices.
While these hardware enhancements in devices and memory management increase performance capabilities during the operation of virtualized systems, they nevertheless, complicate migration tasks. Because device state (e.g., configuration information negotiated with a device driver, etc.) is stored in the hardware device itself rather than in an emulated device, device state cannot be readily captured at a source host computer and transferred to a destination host computer during VM migration. To address this issue, current migration methods unmap or otherwise remove pass-through devices from the VM, resulting in “tear-down” and unloading of corresponding device drivers from the guest OS. When the VM is migrated to a destination host computer, pass-through devices at the destination host computer are mapped into the VM, and corresponding device drivers are reloaded into the guest OS. The process of tearing down a device driver at a source host computer and reloading it at a destination host computer can consume a lot of time, resulting in longer periods of service interruption during VM migration.